The present invention relates to an image processing apparatus, image processing method, image sensing apparatus, control method for the image sensing apparatus, and a memory and, more particularly, to an image processing apparatus and method which improve a defective image (degraded image), image sensing apparatus suitably combined with the image processing apparatus, control method for the image sensing apparatus, and memory for controlling the execution of the methods. In this specification, an image called a degraded image indicates a poor-quality image that is out of focus or blurred due to inappropriate movement of a camera or inappropriate image sensing conditions such as an exposure instead of indicating an image having undergone a change from a good image to a degraded image due to a change in quality over time or the like.
As methods of improving a degraded image, e.g., an out-of-focus image or blurred image, into an image with little degradation (ideal image), methods using a Wiener filter, general inverted filter, projection filter, and the like are available. To use these methods, a degradation function must be determined first. An ideal method of determining such a degradation function is a method of analytically obtaining a degradation function from physical factors such as image sensing conditions or estimating a degradation function on the basis of an output from a measuring device (e.g., an acceleration sensor) mounted in an image sensing apparatus.
A degradation function will be described below. The relationship between an ideal image f(x, y), a degraded image g(x, y), a degradation function h(x, y, xxe2x80x2, yxe2x80x2), and random noise on output image xcexd(x, y) is expressed as
∫∫h(x,y,xxe2x80x2,yxe2x80x2)f(xxe2x80x2,yxe2x80x2)dxxe2x80x2dyxe2x80x2+xcexd(x,y)xe2x80x83xe2x80x83(1) 
If an image having a degraded point is not located at the point, except for translation, a point spread function (PSF) is expressed by h(xxe2x88x92xxe2x80x2, yxe2x88x92yxe2x80x2), and mathematical expression (1) is rewritten into
∫∫h(xxe2x88x92xxe2x80x2,yxe2x88x92yxe2x80x2)f(xxe2x80x2,yxe2x80x2)dxxe2x80x2dyxe2x80x2+xcexd(x,y)xe2x80x83xe2x80x83(2) 
If there is no noise, a Fourier transform of the two sides of mathematical expression (2) is performed, and a convolution theorem is applied to the resultant expression, equation (3) is obtained:
G(u,v)=H(u,v)F(u,v)xe2x80x83xe2x80x83(3) 
where G(u, V) H(u, v), and F(u, V) are the Fourier transforms of g(x, y), f(x, y), and h(x, y).
H(u, v) is the transfer function of a system for transforming the ideal image f(x, y) into the degraded image g(x, y).
A degradation model in degradation (blur) due to a relative movement between a camera and a scene (object) will be described below as an example. Assume that an image on the image sensing element of the camera remains unchanged over time except this relative movement. If the relative movement is approximately equal to the movement of the image sensing element in the same plane, the total exposure light amount at one point on the image sensing element can be obtained by integrating an instantaneous exposure light amount with respect to an exposure time. Assume that the time required to open/close the shutter can be neglected. Letting xcex1(t) and xcex2(t) be the x and y components of the displacement, equation (4) can be established:
g(x,y)=∫xe2x88x92T/2T/2f(xxe2x88x92xcex1(t),yxe2x88x92xcex2(t))dtxe2x80x83xe2x80x83(4) 
where T is the exposure time, and the integration range is set from t=xe2x88x92T/2 to t=T/2 for the sake of convenience.
A Fourier transform of the two sides of equation (4) yields equation (5):                               G          ⁡                      (                          u              ,              v                        )                          =                              ∫                                          ⅆ                x                            ⁢                              ∫                                                      ⅆ                    y                                    ⁢                                      xe2x80x83                                    ⁢                                      exp                    ⁡                                          [                                                                        -                          j2                                                ⁢                                                  xe2x80x83                                                ⁢                                                  π                          ⁡                                                      (                                                          ux                              -                              vy                                                        )                                                                                              ]                                                        ⁢                                                            ∫                                                                        -                          T                                                /                        2                                                                    T                        /                        2                                                              ⁢                                          xe2x80x83                                        ⁢                                          ⅆ                                              tf                        ⁡                                                  (                                                                                    x                              -                                                              α                                ⁡                                                                  (                                  t                                  )                                                                                                                      ,                                                          y                              -                                                              β                                ⁡                                                                  (                                  t                                  )                                                                                                                                              )                                                                                                                                                  ⁢                      
                    =                                    ∫                                                -                  T                                /                2                                            T                /                2                                      ⁢                          xe2x80x83                        ⁢                                          ⅆ                t                            ⁢                              ∫                                                      ⅆ                    x                                    ⁢                                      ∫                                                                  ⅆ                                                  yf                          ⁡                                                      (                                                                                          x                                -                                                                  α                                  ⁡                                                                      (                                    t                                    )                                                                                                                              ,                                                              y                                -                                                                  β                                  ⁡                                                                      (                                    t                                    )                                                                                                                                                        )                                                                                              ⁢                                              exp                        ⁡                                                  [                                                                                    -                              j2                                                        ⁢                                                          xe2x80x83                                                        ⁢                                                          π                              ⁡                                                              (                                                                  ux                                  -                                  vy                                                                )                                                                                                              ]                                                                                                                                                                            (        5        )            
If xxe2x88x92xcex1(t)="xgr" and yxe2x88x92xcex2(t)=xcex7, equation (5) is rewritten into equation (6):                               G          ⁡                      (                          u              ,              v                        )                          =                              ∫                                          ⅆ                x                            ⁢                              ∫                                  ∫                                                            ⅆ                      ξ                                        ⁢                                          ⅆ                      η                                        ⁢                                          xe2x80x83                                        ⁢                                          f                      ⁡                                              (                                                  ξ                          ,                          η                                                )                                                              xc3x97                                          exp                      ⁡                                              [                                                                              -                            j2                                                    ⁢                                                      xe2x80x83                                                    ⁢                                                      π                            ⁡                                                          (                                                                                                u                                  ⁢                                                                      xe2x80x83                                                                    ⁢                                  ξ                                                                -                                                                  v                                  ⁢                                                                      xe2x80x83                                                                    ⁢                                  η                                                                                            )                                                                                                      ]                                                              ⁢                                          exp                      ⁡                                              [                                                                              -                            j2                                                    ⁢                                                      xe2x80x83                                                    ⁢                                                      π                            ⁡                                                          (                                                                                                                                    α                                    ⁡                                                                          (                                      t                                      )                                                                                                        ⁢                                  u                                                                +                                                                                                      β                                    ⁡                                                                          (                                      t                                      )                                                                                                        ⁢                                  v                                                                                            )                                                                                                      ]                                                                                                                          ⁢                      
                    =                                                    F                ⁡                                  (                                      u                    ,                    v                                    )                                            ⁢                                                ∫                                                            -                      T                                        /                    2                                                        T                    /                    2                                                  ⁢                                                      exp                    ⁡                                          [                                                                        -                          j2                                                ⁢                                                  xe2x80x83                                                ⁢                                                  π                          ⁡                                                      (                                                                                          u                                ⁢                                                                  xe2x80x83                                                                ⁢                                                                  α                                  ⁡                                                                      (                                    t                                    )                                                                                                                              +                                                              v                                ⁢                                                                  xe2x80x83                                                                ⁢                                                                  β                                  ⁡                                                                      (                                    t                                    )                                                                                                                                                        )                                                                                              ]                                                        ⁢                                      ⅆ                    t                                                                        =                                          F                ⁡                                  (                                      u                    ,                    v                                    )                                            ⁢                              H                ⁡                                  (                                      u                    ,                    v                                    )                                                                                        (        6        )            
According to equation (6), the degradation is modeled by equation (3) or mathematical expression (2) which is equivalent to equation (3). The transfer function H(u, v) for this degradation is given by
H(u, v)=∫xe2x88x92T/2T/2 exp[xe2x88x92j2xcfx80(uxcex1(t)+vxcex2(t))]dtxe2x80x83xe2x80x83(7) 
In this case, if camera shake occurs in a direction at an angle xcex8 with respect to the x-axis at a predetermined speed V for a time T, a point response function is given as                               H          ⁡                      (                          u              ,              v                        )                          =                              sin            ⁢                          xe2x80x83                        ⁢            πω            ⁢                          xe2x80x83                        ⁢            T                    πω                                    (        8        )            
where xcfx89 is given by equation (9)
xcfx89xe2x80x94(uxe2x88x92uo)V cos xcex8+(vxe2x88x92vo)V sin xcex8xe2x80x83xe2x80x83(9) 
where uo and vo are the center coordinates of the image. When xcfx89 is minimum, H(u, v)=T is approximately established.
Likewise, a degradation model of degradation due to a blur can be expressed by a function. Assume that a phenomenon of blurring is based on a normal (Guassian) distribution rule. In this case, letting r be the distance from a central pixel and "sgr"2 is an arbitrary parameter in the normal distribution rule, a degradation function h(r) is given by                               h          ⁡                      (            r            )                          =                              1                          σ              ⁢                                                2                  ⁢                  π                                                              ⁢                      exp            ⁡                          (                              -                                                      r                    2                                                        σ                    2                                                              )                                                          (        10        )            
Processing for improving a degraded image using an inverted filter will be described next. Assume that the degraded image g(x, y) and the ideal image f(x, y) are based on the model expressed by mathematical expression (2). If there is no noise, the Fourier transforms of g(x, y), f(x, y), PSF, and h(x, y) satisfy equation (3). In this case, equation (3) is modified into
F(u,v)=G(u,v)/H(u,v)xe2x80x83xe2x80x83(11) 
According to equation (11), if H(u, v) is known, the ideal image f (x, y) can be improved by multiplying the Fourier transform G(u, v) of the degraded image by 1/H(u, v) and performing an inverse Fourier transform of the product. In other words, the transfer function of the filter is 1/H(u, v).
In practice, the application of equation (3) poses various problems. For example, in consideration of noise, mathematical expression (2) can be written into
G(u,v)=H(u,v)F(u,v)+N(u,v)xe2x80x83xe2x80x83(12) 
where N(u, v) is the Fourier transform of xcexd(x, y).
According to equation (12), when the filter (1/H) (u, v)) is applied to the Fourier transform of the degraded image, equation (13) is established:                                           G            ⁡                          (                              u                ,                v                            )                                            H            ⁡                          (                              u                ,                v                            )                                      =                              F            ⁡                          (                              u                ,                v                            )                                +                                    N              ⁡                              (                                  u                  ,                  v                                )                                                    H              ⁡                              (                                  u                  ,                  v                                )                                                                        (        13        )            
Consider a system in which the degraded image recorded by the digital camera is loaded into an information processing apparatus by an image receiving unit controlled by a TWAIN driver or the like, and the degraded image is improved to generate an ideal image. In this case, a technique of determining a degradation function obtained by modeling the process of generating a degraded image, and improving the degraded image by using an image improving algorithm generally called deconvolution using a Wiener filter or the like is considered as the most effective improving technique.
In such a conventional technique, however, since no consideration is given to degradation parameters (e.g., a shake direction, shake speed, and the like if degradation is caused by camera shake) required to determine a degradation function and image sensing conditions (an exposure time, exposure light amount, and the like), a sufficient improving effect for a degraded image cannot be obtained.
Many studies have been made on techniques of obtaining a degradation parameter by estimating a degradation state from the image feature amount (e.g., an auto-correlation function) of a degraded image. A method of estimating a line spread function, edge spread function, or the like instead of the above point spread function is also available. For example, the following method of determining a degradation function is known. In this method, if an acute edge portion is present in an original image, the edge portion is differentiated to obtain a line spread function, and a degradation function is determined by using an image reconstruction method.
If, however, an image is to be improved by using an image improving algorithm using a Wiener filter or the like, only several percent of an error included in a parameter will cause large noise to be superimposed on an improved image. For example, in the technique of obtaining a degradation parameter by estimating a degradation state from the image feature amount (e.g., an auto-correction function) of a degraded image, since the parameter often includes a large error, the improving effect is low. Even if a measuring device (acceleration sensor or the like) mounted in an image sensing apparatus such as a camera is to be used, since it is technically very difficult to suppress an error within several percent, the problem of a parameter error cannot be neglected.
As described above, in the prior art, no consideration is given to degradation parameters, image sensing conditions, and the like which are required to determine a degradation function, and a degradation parameter cannot be accurately estimated. For this reason, even if a degraded image is improved, the improving effect is not sufficient.
In the technique of using an edge spread function to estimate a degradation parameter, it is difficult to extract an edge portion, because the original image is degraded. The present applicant has proposed a technique (Japanese Patent Laid-Open No. 7-121703) of designating a small area, of a degraded image, which includes an edge portion, improving the image within the small area while changing a parameter of a degradation function, obtaining the degree of image improvement corresponding to the parameter, and improving the entire image by using a parameter with which the degree of image improvement becomes the highest. This proposal, however, can be applied to only estimation of an edge spread function but cannot be applied to estimation of a point spread function.
An edge portion of an original image corresponds to a high-frequency component in terms of frequency. When the image is to be improved by using various filters, the edge portion is an area on which the largest noise is superimposed. Even if, therefore, a parameter with which the degree of image improvement becomes the highest is obtained, the possibility of a low precision due to noise is high. In addition, it is cumbersome to designate a small area, of an original image, which includes an edge portion every time improving operation is performed.
In general, in a system including a digital camera, an image is loaded into an information processing apparatus by an image receiving unit controlled by a TWAIN driver or the like, and the resultant image is output to an image output apparatus (e.g., a printer or display). In this case, the image processing controlled by the TWAIN driver or the like generally includes gamma correction, color conversion, and the like. However, gamma correction, color conversion, and the like belong to a nonlinear system and contradict an LTI (Linear Time-Invariant) system serving as a precondition for the above image improving algorithm. That is, when a degraded image having undergone gamma correction or color conversion is improved, very large noise is superimposed on the improved image.
Consider a system for converting a degraded image recorded on a silver halide film into an electronic image by using an image input device such as a film scanner, and improving the degraded image by arithmetic operation. In general, such a system also uses a technique of forming a degradation function by modeling the process of generating a degraded image and improving the image by using an image improving algorithm generally called deconvolution using a Wiener filter or the like. This technique is regarded as the most effective improving technique.
In practice, however, the electronic image information has been affected by the aberrations of the lenses of the image sensing apparatus, and the luminance value has been converted by the characteristics (ISO, film xcex3, and the like) of the film itself, the gamma characteristics of the photoelectric conversion system of the image input device such as a film scanner, and the like.
The above lens aberrations, film characteristics, and scanner characteristics (so-called scanner xcex3 and the like) also generally belong to a nonlinear system and contradict an LTI (Linear Time-Invariant) system as a precondition for the image improving algorithm described above. That is, when the degraded image obtained in the above manner is improved by deconvolution, very large noise is superimposed on the improved image.
Even if an extremely ideal improved image is generated, noise superimposed on the improved image cannot be completely eliminated. In addition, since the improved image has no correlation with the color gradation and luminance distribution of the original image, the improved image cannot become an image that is worth looking at without any processing.
The present invention has been made in consideration of the prior art, and has as its object to provide an image processing method and apparatus for improving an image with a high precision while keeping the computation load low in improving an image using various degradation functions including degradation parameters even if no estimated value is available for each degradation parameter or an error is included in the degradation parameter obtained by analytical estimation or from an output from a measuring device mounted in the image sensing apparatus.
The present invention has been made in consideration of the above situation, and has as its object to obtain a high-quality improved image from a degraded image.
It is another object of the present invention to provide an image improving system and method which improve a degraded image by using degradation information required to increase the improving effect while suppressing noise to a minimum, and perform image processing for the image to finally obtain an image that is worth looking at.
It is still another object of the present invention to provide an image improving method and system for improving an image with a high precision while keeping the computation load low in improving an image using various degradation functions including degradation parameters even if no estimated value is available for each degradation parameter or an error is included in the degradation parameter obtained by analytical estimation or from an output from a measuring device mounted in the image sensing apparatus.
In one aspect, the present invention relates to an image processing method, apparatus and computer program for improving an image sensed by an image sensing apparatus and processed according to a first conversion process. The method, apparatus and computer program involve determining whether or not the first conversion process includes a nonlinear conversion, processing the image according to a second conversion process inverse to the first conversion process if the first conversion process includes the nonlinear conversion, and processing the image processed according to the second conversion process, according to a function for improving the image.
According to an aspect of the present invention, there is provided an image processing method of improving an image sensed by an image sensing apparatus, comprising:
the input step of inputting image data representing the image through input means;
the inversion step of performing, for the image, processing inverse to conversion processing performed for the input image data; and
the improving step of performing improving processing for the image having undergone the inversion processing on the basis of a degradation function in image sensing operation.
Preferably, the inversion step further comprises the acquisition step of acquiring information on the conversion processing performed for the input image data.
The acquisition step preferably comprises looking up a table in which a type of the input means is made to correspond to a conversion processing method used by the input means, and acquiring information on the conversion processing performed for the image data on the basis of the type of the input means.
The improving step preferably comprises obtaining a degradation function on the basis of characteristic information on the image sensing apparatus, and performing improving processing for the image data on the basis of the degradation function.
The improving step preferably comprises looking up a table in which a type of an image sensing apparatus for sensing an image is made to correspond to characteristics of the apparatus, and acquiring the characteristic information on the basis of the type of image sensing apparatus.
The conversion processing is preferably nonlinear conversion processing.
The nonlinear conversion processing preferably includes gray scale conversion.
Preferably, the image data has undergone one of conversion processing including a nonlinear conversion and conversion processing including no nonlinear conversion in accordance with a conversion mode, the input step comprises inputting the conversion mode together with image data, and the inversion step comprises referring to information of the conversion mode and performing inversion processing for the image data if the image data has undergone a conversion including a nonlinear conversion.
Preferably, the input step comprises inputting, together with image data, a degradation parameter which is recorded by the image sensing means together with an image and indicates a physical quantity of an event as a cause of degradation, and the improving step comprises specifying a degradation function on the basis of the degradation parameter, and performing improving processing for the image in accordance with the degradation function.
Preferably, the input step comprises inputting, together with image data, an image sensing condition, which is recorded by the image sensing means together with an image, and the improving step comprises specifying a degradation function on the basis of the image sensing condition and performing improving processing for the image data in accordance with the degradation function.
Preferably, the method further comprises the post-processing step of performing, for the image data improved in the improving step, inversion processing to the inversion processing performed for the image data in the inversion step.
Preferably, the post-processing step further comprises determining a degree of degradation on the basis of the degradation parameter, and performing noise removal processing for the image data if it is determined that the degree of degradation is high.
Preferably, the image sensing apparatus electronically records light from an object to be image-sensed as image data by photoelectric conversion, and the input step comprises inputting the electronically recorded image data.
Preferably, the image sensing apparatus optically records an image on a silver halide film, and the input step comprises inputting the image data by using input means for photoelectrically converting the image on the silver halide film.
Preferably, the image sensing apparatus records characteristic information of the image sensing apparatus and/or an image sensing condition and/or a degradation parameter on a magnetic medium corresponding to a silver halide film, and the input step comprises inputting the pieces of information together with image data.
According to the second aspect of the present invention, there is provided an image processing method of improving an image having suffered degradation in image sensing operation by using a degradation function for quantifying the degradation, comprising:
the first improving step of improving the image by using the degradation function while changing a degradation parameter included in the degradation function;
the improvement degree calculation step of obtaining a degree of image improvement of the improved image; and
the second improving step of selecting a degradation parameter with which the degree of image improvement becomes highest, and generating an improved image by a degradation function to which the parameter is applied.
The improving step preferably comprises improving the image while changing a value of one parameter at a time, and repeatedly improving the image while changing a value of a next parameter after a value of one parameter with which the degree of image improvement become highest is determined.
Preferably, the method further comprises the segmentation step of segmenting the image, the first improving step comprises improving the image with respect to an image area selected from segmentated areas, and the second improving step comprises performing improving processing for the entire image.
Preferably, the method further comprises the selection step of selecting an area, of the areas segmented in the segmentation step, in which a center of gravity of a frequency distribution corresponds to a highest frequency, and the first improving step comprises improving the area selected in the selection step.
Preferably, the method further comprises the input step of inputting a value of a degradation parameter, and the first improving step comprises improving the image by using the degradation function while changing a value of a degradation parameter included in the degradation function within a predetermined range including a value input in the input step.
Other features and advantages of the present invention will be apparent from the following description taken in conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts throughout the figures thereof.